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(57) In a preferred embodiment, an apparatus for 
eye tracl<ing, incfuding: an illuminator; reflection appa- 
ratus to reflect illumination from the illuminator onto a 
surface of a windshield of a vehicle in which the wind- 
shield is installed, such that the illumination is reflected 



onto at least an eye of a person in the vehicle, and to 
reflect an image of the at least an eye; and a sensor to 
receive a reflection of the Image of the at least an eye 
reflected by the reflection apparatus and to produce an 
output signal representative thereof. 
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Description 

BACKGROUD OF THE INVENTION 
Field of the Invention 

[0001] The present invention relates to eye tracking 
systenns generally and, more particularly, but not by way 
of limitation, to a novel eye tracking system using, in 
part, an existing head up display (HUD). 

Background Art 

[0002] While the present invention is described gen- 
erally in the context of determining inattentiveness of the 
driver of a vehicle, it will be understood thatthe invention 
is application, as well, to any situation in which tracking 
of eye movements is desirable. Communities in which 
eye tracking is used include the medical, research, and 
academic communities where eye tracking is useful, for 
example, In determining psychological response and 
how a person scans. 

[0003] A large number of the approximately 35,000 
vehicular deaths and a much larger number of vehicular 
accidents each year in the United States can be attrib- 
uted to driver inattention. Driver inattention may result 
from drowsiness, use of a telephone, use of a computer, 
changing stations on a radio, conversing with another 
occupant of the vehicle, and similar causes. Whatever 
the cause, such driver inattention may result in fatalities 
and non-fatal acddents. 

[0004] In response to this problem, systems have 
been developed to continuously track an eye of a driver 
and use this information to determine when the driver 
becomes inattentive and to institute measures to alert 
the driver of the potential disaster. Eye tracking can be 
an accurate, non-intrusive, non-contact method, which 
detects driver Inattention and fatigue by recording eye 
movement. Parameters such as blink duration, blink fre- 
quency, and eye closure can be useful for detecting 
drowsiness, fatigue, and other causes of inattentive- 
ness. Pupillary data such as diameter, shape, location, 
and rate of change can be useful in detemnining the di- 
rection of the driver's gaze and workload. Thus, it Is a 
key safety system that can be used to determine the 
state of the driver for workload management systems. 
[0005] Although there are a number of eye tracking 
methods, many are quite expensive and many involve 
undesirable intrusiveness, both undesirable features for 
a vehicular system. Some of these methods are dis- 
cussed below. 

[0006] The limbus eye tracking method utilizes the dif- 
ference in reflectance between the sclera and the iris. 
The reflectively difference produces a contrast that can 
be monitored by photodetectors mounted on either side 
of the eyeball. This method requires head mounted sen- 
sors and, when used alone, is sensitive to head move- 
ment when calculating gaze. 



[0007] The pupil tracking method is similar to the lim- 
bus tracking method, except that it uses the boundary 
between the pupil and iris, rather than the boundary be- 
tween the sclera and the iris. This method is advanta- 

s geous in that the pupil is far less covered by the eyelid 
than is the limbus and the border is sharper than that of 
the limbus, thus offering a higher resolution. A disad- 
vantage is that the contrast is lower. 
[0008] The cornea/pupil relationship method may be 

10 one of two types, in dark pupil tracking, coilimated, on- 
axls IR illumination causes a dark pupil to appear bright 
The high contrast with the dark pupil acts as a light sink. 
In bright pupil tracking, uncoilimated, off -axis IR illumi- 
nation reflects off the retina similar to the reflection seen 

15 from the eyes of a nocturnal animal or in red-eye from 
flash photography. In either case, a CCD or CMOS de- 
tector is used. Pupil edge Is located and the center Is 
calculated. 

[0009] The artificial neural network method uses brain 
20 waves and artificial intelligence in software to calculate 
alertness, etc. Capturing the brain waves requires the 
use of electrodes placed on the scalp or placing the sub- 
ject in an MR! type of device. 

[0010] The dual Purkinje image method compares the 

25 comeal reflection to the reflection from the back surface 
of the lens. IV^easurements are made of the relative dis- 
placement between the first and fourth reflections, to the 
shape and size of the pupil, which represent the focal 
point. Because the IR illumination has to travel through 

30 seven interfaces, the method is very Inefficient and re- 
quires a relatively powerful Illumination source. 
[0011] The electro-oculographic method calculates 
eye position by measuring the potential difference be- 
tween the front and back of the eyeball. This method is 

35 simple, but not reliable, due to signal variations over 
time, and not precise, due to noise levels. Of course, 
electrodes must be placed around the eye. 
[0012] The search coil method requires that induction 
coils be embedded in the sclera or In tight fitting contact 

40 lenses. This is a very accurate method, but also quite 
intrusive. 

[0013] The video-oculographic method gives an ac- 
curate measurement of eye components, including oc- 
ular torsion. The method also provides pupil diameter 
<s measurement and monocular and binocular measure- 
ment with pupil diameter. There is no drift and reduced 
noise sensitivity. 

[0014] As noted above, the foregoing methods are un- 
suitable for vehicular use because of cost and/or intru- 
30 siveness, although some produce fairiy accurate re- 
sults. 

[0015] Commonly, a system used in vehicular eye 
tracking includes one or two IR sources directed at an 
eye of the driver, a combination of mirrors and motors, 
55 an eye camera module, and a processing and control 
unit. As the two light sources shine on an eye, two spots 
(invisible, but seen by the infrared camera) occur in the 
pupil. The eye camera module captures and sends an 
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image of the eye to the processing and control unit, 
which calculates the direction of driver gaze by connpar- 
ing the locations of the two spots in the pupil. The mirror/ 
motor combination is used in front of the eye camera 
adjusts to try to track the driver's eye as the driver's head 
is moved and also adjusts to driver height through cali- 
bration. This requires a complex and expensive auto fo- 
cus and tracking system. 

[0016] One problem is that such current eye tracking 
systems use a small field of view lens to cover a small 
area around the eye of the driver and an image sensor 
having a sub-QVGA ("quarter video graphic array) res- 
olution (or one-quarter of the standard 640x480 VGA 
pixelation). Because of these limitations, conventional 
eye tracking systems permit only limited head move- 
ment. If the limited degree of head movement is exceed- 
ed by sudden head movement, head rotation, or vibra- 
tion, for example if the vehicle runs on a rocky road, the 
system will "lose" the eye and becomes ineffective until 
the eye can again be tracked which can take some time. 
[0017] Another problem with such current eye track- 
ing systems is that, since each person has a slightly dif- 
ferent ocular profile that can result in a large error in cal- 
culation, the eye tracking system needs to be calibrated 
for each different driver. This is done by having the driver 
sit stIN in the driver's seat, while an instructor sits in the 
front passenger seat. The instructor tells the driver to 
look at a particular spot, or fixate on predetermined 
aHgnment coordinates, for a short period of time. The 
instructor then measures and records oculometric data. 
This procedure is repeated some six or nine times. 
Then, the instructor takes the recorded data and devel- 
ops an ocular profile of the driver that is recorded in a 
look-up table that Is manually accessed for each driver 
This must be done for each potential driver of the vehi- 
cle. In addition to the fairly tedious start-up procedure, 
periodic recalibration is required because of drift. 
[0018] In addition to the above problems, the use of 
electromechanical components inherently renders a 
system complex and relatively costly and introduces ad- 
ditional areas in which alignment is critbal and In which 
the wearing of mechanical linkages eventually introduc- 
es additional errors into the eye tracking system. 
[0019] A further disadvantage of conventional eye 
tracking systems Is that placement of a detector in a ve- 
hicle becomes a problem. One reason for this is that the 
path from the drivers' eye to the detector cannot become 
obstructed, for example, by the driver's hands. It would 
also be desirable to utilize some of existing systems in 
a vehicle. It is also desirable that the driver's field of view 
not be obstructed. 

[0020] Accordingly, it is a principal object of the 
present invention to provide an eye tracking system that 
has no moving parts. 

[0021] It is a further object of the invention to provide 
such an eye tracking system that can increase the field 
of view to well beyond the area around the eye, while 
maintaining adequate resolution. 



[0022] It is an additional object of the invention to pro- 
vide such an eye tracking system that is automatically 
calibrated for each driver. 

[0023] It is another object of the invention to provide 
5 such an eye tracking system that permits nearty 180^ 
head rotation, while continuing to monitor an eye. 
[0024] It is yet a further object of the invention to pro- 
vide such an eye tracking system that includes automat- 
ic driver identification. 
10 [0025] It is yet an additional object of the invention to 
eliminate or greatly reduce the potential for "lost" eye 
contact. 

[0026] It is yet another object of the invention to pro- 
vide such an eye tracking system that uses high reso- 
15 lution methods to examine the entire face of a driver and 
retrieve and ocular profile based on the identification of 
the driver or to prepare and store an ocular profile, using 
a high resolution image of an eye, if the driver is not 
recognized. 

[0027] Yet a further object of the invention is to provide 
such an eye tracking system that utilizes portions of an 
existing system in a vehicle. 

[0028] Yet an additional object of the invention is to 
provide such an eye tracking system that does not ob- 
struct the driver's field of view. 

[0029] Other objects of the present invention, as well 
as particularfeatures, elements, and advantages there- 
of, will be elucidated in, or be apparent from, the follow- 
ing description and the accompanying drawing figures. 

SUMMARY OF THE INVENTION 

[0030] The present invention achieves the above ob- 
jects, among others, by providing, In a preferred embod- 
iment, an apparatus for eye tracking, comprising, an ap- 
paratus for eye tracking, comprising: an illuminator; re- 
flection means to reflect illumination from said illumina- 
tor onto a surface of a windshield of a vehicle in which 
said windshield is installed, such that said illumination 
is reflected onto at least an eye of a person in said ve- 
hicle, and to reflect an image of said at least an eye; and 
a sensor to receive a reflection of said image of said at 
least an eye reflected by said reflection means and to 
produce an output signal representative thereof. 

BRIEF DESCRIPTION OF THE DRAWING 

[0031] Understanding ofthe present invention and the 
various aspects thereof will be facilitated by reference 
to the accompanying drawing figures, provided for pur- 
poses of Illustration only and not intended to define the 
scope ofthe invention, on which: 

Figure 1 is schematic view of an eye tracking sys- 
tem according to the present invention used to track 
the eye movements of a person. 
Figure 2 is a partially schematic view of a dashboard 
and windshield of a vehicle in which the present in- 
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vention is installed. 

Figure 3 shows the reflections of IR sources In an 
eye. 

Figure 4 shows a full face view and a view of an eye 
as used in the present invention. 
Figure 5 shows various features that may be used 
by the present invention to identify a driver. 
Figure 6 is a logic flow diagram of the system of the 
present Invention. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0032] Reference should now be made to the drawing 
figures on which similar or identical elements are gh^en 
consistent identifying numerals throughout the various 
figures thereof, and on which parenthetical references 
to figure numbers direct the reader to the view(s) on 
which the elemenl(s) being described is (are) best seen, 
although the element(s) may be seen on other figures 
also. 

[0033] Figure 1 illustrates the present invention incor- 
porated in an head up display (HUD) system. The con- 
ventional elements of the HUD include an engine 20 
having a backlight source 22, an optical coupler 24, a 
polarizer/substrate 26, a diff user 28, an a display device 
30 having thereon a polarizing optical film 32, ail the el- 
ements being arranged as shown on Figure 1 . HUD en- 
gine 20 projects an image that is reflected by an aspher- 
Ic mirror 40 onto a vehicle windshield 42. This image 
may include any Information that is to be Imparted to the 
driver of the vehicle, such as vehicle speed, distance to 
orf rom a location, etc. The Image is viewed by the driver 
without obstruction of the driver's view through the wind- 
shield. 

[0034] The present invention adds the elements of an 
IR sensor/camera 50 that provides an input to a 
processing platfomn 52, a ring of IR illuminators 54 dis- 
posed near the IR sensor/camera 50, and an IR mirror 
56 that reflects radiation from the IR illuminators onto 
aspheric mirror 40. Aspheric mirror 40, In turn reflects 
the radiation onto windshield 42 which may have an IR 
reflection enhanced surface 58. The radiation reflects 
off surface 58 and showers an eye 60 and the face (not 
shown on Figure 1) of the driver. The resulting Image of 
the eye and face of the driver is reflected In sequence 
off surface 58, aspheric mirror 40, and IR mirror 56 and 
is received by IR sensor/camera 50. The signal from 
sensor/camera 50 is passed to processing and control 
circuitry in processing platfomn 52 and used in the man- 
ner described below, the processing and control circuitry 
also controlling the other elements of the system. IR 
sensor/camera 50 could include electronic pan tilt to 
compensate for head and eye movement. Illumination 
sources other than IR may be provided, as long as the 
other illumination sources are non-intrusive. The system 
10 has no moving parts. 

[0035] Figure 2 illustrates some of the elements 



shown on Figure 1 installed In a dashboard 62 of a ve- 
hicle (not otherwise shown), including I R sensor/camera 
50, IR illuminators 54, IR mirror 56, HUD engine 20 (in 
a HUD unit 64). Figure 2 also shows an area 64 repre- 

5 senting the HUD "eyebox" and an area 66 representing 
a larger area that Is the area of the field of detection of 
the driver's gaze or facial features. It will be understood 
that the optical elements of the present invention can be 
disposed such that there is little chance of obstruction 

10 of the line of sight between the optical elements. Also, 
the system is non-Invasive. 

[0036] The conventional, low-resolution CCD sensor 
in sensor/camera 50 has been replaced in this invention 
with a high-resolution CMOS device, which permits a 

IS larger field of view, while maintaining the minimum re- 
quired pixel density for precise oculometric data acqui- 
sition and pemilts increased freedom of head move- 
ment. This change increases the resol ution of the optical 
from the conventional sub-QVGA to high resolution 

20 such as SXGA (1280 x 1 024 pixels). Now, the entire face 
of the user can be viewed, permitting the use of known 
techniques to extract facial features after suitable image 
processing. This results in the 8-10 pixels per degree 
that is achieved with conventional systems that view on- 

25 ly the eye of the driver. 

[0037] The facial feature extraction enables the sys- 
tem to recognize the driver of whom the ocu lometric pro- 
file is stored and uploaded quickly This then eliminates 
the need for annoying recallbration for different learned 

30 drivers, as well as expedites the learning process. Facial 
feature extraction also enables auto-calibration by the 
use of search and recognition algorithms to detemnined 
eye location and its associated oculometric geometry 
within the system. Facial feature extraction can also be 

35 used to determine how the vehicle can be operated or 
access to various areas of the vehicle. For example, if 
the driver is not recognized, the operating system may 
go into "valet mode" in which certain areas of the vehi- 
cle, such as the glove compartment or the trunk of the 

40 vehble cannot be accessed by the driver. 

[0038] Figure 3 illustrates the reflections 68 of I R illu- 
minators 54 (Figure 1 ) in the eye 60 of a driver. It will be 
understood that the reflections shown on Figure 3 are 
for illustrative purposes only and that the reflections 

45 would not be visible. This is an example of a non-contact 
technique that uses the pupil/cornea reflection principle. 
The positions of the pupil and reflex centers are deter- 
mined from a digitized video image and the driver's point 
of gaze is determined by calculating the difference vec- 

so tors of reflections 68. 

[0039] Figure 4 illustrates the full face view and an eye 
view as used by the system, the use of which is de- 
scribed below. 

[0040] Figure 5 illustrates facial features that may be 
55 used to Identify a driver. These include the positions and 
sizes of: the eyes 70, the comers 80 of the eyes, the 
ears 90, the corners 1 00 of the mouth, the chin 1 1 0. and 
the ala 120, of the nose. 
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[0041 ] Initial calibration now requires only that a driver 
sit in the driver's seat. The system automatically scans 
the face of the driver and stores identifying information 
in memory. Now, when the same driver sits in the driver's 
seat, the system scans the face of the driver, recognizes 5 
the driver and automatically uploads the driver's profile, 
einninating the need to re-calibrate for different drivers. 
This profile can also be used to Identify the driver and 
load stored presets. Of course, portions of the system 
may be overridden if, for example, the vehicle is to be io 
given to a parking valet. 

[0042] Figure 6 is a logic flow diagram illustrating the 
operation of the system. 

[0043] First, using SXGA resolution, the whole face of 
a driver constitutes the field of view. The driver enters 
the vehicle at step 200. At step 21 0. the system search- 
es for particular facial features. At step 220, the system 
acquires the facial features of the driver (Figure 5). At 
step 230, the system determines if the driver is recog- 
nized. If the driver is recognized, the driver's ocular pro- 
file is retrieved at step 240 from profile storage 250, This 
is used at step 260 to perform oculometric calculations. 
At step 270, oculometrics are determined while contin- 
uing to use the whole face of the driver in the field of 
view. 25 
[0044] If the driver is not recognized at step 230, the 
system uses SXGA resolution in which an eye of the 
driver fills the entire field of view and, at step 280, the 
system searches for ocular features. At step 290, the 
system acquires the ocular features. At step 300, the so 
system calibrates and creates an ocular profile. The oc- 
ular profile created at step 300 is then categorized at 
step 310 with the facial features acquired at step 220 
and the ocular profile is stored in the profile storage 250. 
[0045] At the foregoing steps are automatic and re- 35 
quire no manual input on the part of the driver or any 
other person. 

[0046] Should eyecontact be "lost", by the driver com- 
pletely moving the driver's head, for example, the sys- 
tem goes into standby mode until a face is recognized, 40 
with the system using the whole face In the field of view. 
Then, knowing where the eye being tracked is located 
on the face, the system can again use the ocular profile 
to perform oculometric calculations. The system differ- 
entiates between a normal blink and the loss of data. 45 
[0047] In the embodiments of the present Invention 
described above, it will be recognized that individual el- 
ements and/or features thereof are not necessarily lim- 
ited to a particular embodiment but, where applicable, 
are Interchangeable and can be used In any selected so 
embodiment even though such may not be specifically 
shown. Terms such as "upper", "lower", "inner", "outer", 
"inwardly", "outwardly", "vertical", "horizontal", and the 
like, when used herein, refer to the positions of the re- 
spective elements shown on the accompanying drawing 55 
figures and the present invention is not necessarily lim- 
ited to such positions. 

[0048] It will thus be seen that the objects set forth 



above, among those elucidated in, or made apparent 
from, the preceding description, are efficiently attained 
and, since certain changes may be made in the above 
construction without departing from the scope of the in- 
vention, it is intended that all matter contained in the 
above description or shown on the accompanying draw- 
ing figures shall be interpreted as illustrative only and 
not in a limiting sense. 

[0049] It is also to be understood that the following 
claims are intended to cover all of the generic and spe- 
cific features of the invention herein described and all 
statements of the scope of the invention which, as amat- 
ter of language, might be said to fall therebetween. 



Claims 

1 . An apparatus for eye tracking, comprising: 

(a) an illuminator (54); 

(b) reflection means (40) to reflect illumination 
from said illuminator onto a surface (58) of a 
windshield (42) of a vehicle in which said wind- 
shield (42) is installed, such that said illumina- 
tion is reflected onto at least an eye (60) of a 
person in said vehicle, and to reflect an image 
of said at least an eye (60); and 

(c) a sensor (50) to receive a reflection of said 
image of said at least an eye (60) reflected by 
said reflection means (40) and to produce an 
output signal representative thereof. 

2. An apparatus for eye tracking, as defined in Claim 
1 , wherein: said reflection means (40) is at least part 
of a head up display In said vehble. 

3. An apparatus for eye tracking, as defined in Claim 
1 , further comprising: 

(a) a controller (52) operatively connected to 
said sensor (50) to receive said output signal; 

(b) said controller (52) configured to cause said 
sensor (50) to view an entire face of said person 
to obtain predetemntned facial features (70, 80, 
90, 100, 110) and to produce a signal repre- 
sentative of said facial features (70, 80. 90, 
100, 110); and 

(c) said controller (52) configured to receive 
said signal and, if said person Is recognized 
from said facial features (70, 80, 90, 1 00, 1 1 0), 
to retrieve a stored ocular profile (250) of said 
reflection and to use said ocular profile (250) 
for eye tracking while viewing said entire face 
of said person. 

4. An apparatus for eye tracking, as defined in Claim 
3, wherein: if said person is not recognized, said 
controller (52) causes said sensor (50) to view only 
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said eye (60) of said person and uses such infor- 
mation to create an ocular profile (250) therefor for 
use in eye tracking. 

5. An apparatus for eye tracl<ing, as defined in Claim s 
1, wherein: all elements of said apparatus operate 
automatically. 

6. An apparatus for eye tracking, as defined in Claim 

1, wherein: all elements of said apparatus operate io 
without mechanical linkages. 

7. An apparatus for eye tracking, as defined in Claim 
1, wherein: said apparatus uses SXGA resolution. 

15 

8. An apparatus for eye tracking, as defined in Claim 
3, wherein: if said person is not recognized, said 
controller (52) restricts how a vehicle can be oper- 
ated and/or restricts access to certain areas of said 
vehicle. 20 

9. An apparatus for eye tracking, as defined in Claim 
3, wherein: If contact with said eye is lost, said con- 
troller (52) uses said entire face of said person to 
locate said eye (60). 25 
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